Device for detecting water on a surface and a method for detecting water on a surface

ABSTRACT

A device for identifying water on a surface, including an optical sensor and a processor. 
     The optical sensor is configured to produce a first image of the surface which has a first optical bandwidth within which the water has a first absorption rate, and a second image of the surface which has a second optical bandwidth within which the water has a second absorption rate that is higher than the first absorption rate. 
     The processor is configured to combine the first image and the second image to produce a combined image in which the surface is reduced or eliminated as compared to the water. In addition, the processor is configured to detect water in the combined image.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from German Patent Application No. DE102020203293.9, which was filed on Mar. 13, 2020, and is incorporatedherein in its entirety by reference.

The present application relates to a device for identifying water on asurface, and to a method of identifying water on a surface.

BACKGROUND OF THE INVENTION

Identification of road conditions is essential for safe driving. Moderncars estimate general risks of road conditions with the help oftemperature and control sensors. Several principles for activeidentification of accumulation of water on the road while usingdifferent main features and results are proposed.

On an RGB picture, it is difficult to distinguish a wet spot from adirty spot, for example, since both are simply darker. Although water istransparent, water is easily visible in RGB pictures due to severaleffects that change the path of the light ray. Computer visionalgorithms or object detection algorithms may be applied to RGB picturesto locate anomalies on the road such as, e.g., puddles of water. Theadvantages of this reflection method include the fact that common imagesensors may be used, making this method a low-cost solution, and thatlocalization and classification are possible. While using an RGB camera,the various effects such as polarization and absorption cannot bedetermined, and therefore, the results of the localization andclassification method are highly dependent on the respective case andare unreliable.

Another method describes measuring the backscatter characteristics ofthe surface. The backscatter behavior of a surface is highly dependenton its roughness. Water and ice are smoother than ordinary roadmaterials such as asphalt. A large portion of the horizontally polarizedpart of the light rays is reflected by the water or ice surface, whilethe vertical part of the light rays penetrates the water surface and isscattered on the ground and/or on ice crystals. Using polarizationfilters to measure these backscattering properties of the environmentprovides reference points for a classification assessment. Measuringbackscattering properties is a cheap solution as it allows using commonimage sensors comprising simple polarization filters. In addition, thissolution makes it possible to reliably locate smooth areas such as waterand ice, for example. However, distinguishing between water and ice ispossible only with a known light source or with a polarized lightsource. Yet this method is susceptible to external influences such asextraneous light and/or variations in the subsurface.

The next method presented is spectral absorption measurement of water.Water has a recognizable absorption characteristic and absorbs much moreenergy within the infrared spectrum than within the range of visiblelight. Within the near infrared (NIR) range, water shows distinctabsorption peaks. Some of these absorption peaks exhibit a significantshift when the water begins to form crystals or to freeze. Usingspecific image sensors such as those based on specific semiconductorsfrom the III-V group, for example, for increased spectral sensitivityallows these peaks within the NIR range to be measured, providing asimple method for distinguishing between water, ice, and snow. Anadvantage of this method is, e.g., reliable classification, e.g. betweenwater, ice, snow and dry soil. Disadvantages of this solution are, e.g.,expensive hardware, low resolution and poor localization.

SUMMARY

According to an embodiment, a device for identifying water on a surfacemay have: an optical sensor for producing a first image of the surfacewhich has a first optical bandwidth within which the water has a firstabsorption rate, and a second image of the surface which has a secondoptical bandwidth within which the water has a second absorption ratethat is higher than the first absorption rate; a processor for combiningthe first image and the second image to produce a combined image inwhich the surface is reduced or eliminated as compared to the water, andfor detecting the water in the combined image; and an additional opticalsensor configured to provide to the processor a third image with a firstpolarization filter having a first polarization angle, and to provide tothe processor a fourth image with a second polarization filter having asecond polarization angle, wherein the two polarization filters differfrom each other with regard to their polarization angles, and whereinthe processor is configured to detect an aggregate state of the waterwhile using the third image and the fourth image in the combined image.

According to another embodiment, a means of locomotion may have: theinventive device for identifying water on a surface, and an interface,wherein the interface is configured to alert a driver of the means oflocomotion and/or to influence control of the means of locomotion if thedevice detects a solid state of the water.

According to yet another embodiment, a method of distinguishing a liquidor solid aggregate state of water in a region containing water may havethe steps of: obtaining a first image of the surface which has a firstoptical bandwidth within which the water has a first absorption rate,obtaining a second image of the surface which has a second opticalbandwidth within which the water has a second absorption rate that ishigher than the first absorption rate, combining the first image and thesecond image to obtain a combined image in which the surface is reducedor eliminated as compared to the water, detecting the water in thecombined image, obtaining a third image with a first polarization filterhaving a first polarization angle, obtaining a fourth image with asecond polarization filter having a second polarization angle, whereinthe two polarization filters differ from each other with regard to theirpolarization angles, evaluating the picture elements of the regionscontaining water of the third and fourth images, and detecting theaggregate state of the water on the basis of the evaluation of thepicture elements.

According to still another embodiment, a non-transitory digital storagemedium may have a computer program stored thereon to perform theinventive method, when said computer program is run by a computer.

Embodiments of the invention provide a device for identifying water on asurface which comprises an optical sensor and a processor.

The optical sensor is configured to provide a first image of the surfacewhich has a first optical bandwidth within which the water has a firstabsorption rate, and a second image of the surface which has a secondoptical bandwidth within which the water has a second absorption ratethat is higher than the first absorption rate.

The processor is configured to combine the first image and the secondimage to produce a combined image in which the surface is reduced oreliminated as compared to the water.

In addition, the processor is configured to detect water in the combinedimage.

Examples of the present invention are based on the finding that relativemeasurements are preferred over absolute measurements so as to reducethe effects of external influencing factors such as the environmentand/or lighting, for example, on the measurement of physical properties.In this case, a physical parameter, e.g., absorption, is extracted fromseveral measurements performed, e.g., by using several optical filters.

Thus, the device provides reliable identification of water on thesurface.

To take advantage of the absorption characteristics of water, the devicetakes two pictures of the same surface by using an optical sensorcomprising bandpass filters for the spectral ranges having high and lowwater absorption rates, respectively.

Other materials exhibit constant absorption rates within the specificspectral range, only water will produce differences. Externalinfluencing factors such as changes in the subsurface or illuminationwill influence both pictures equally.

To reduce or eliminate background, in embodiments the processor isconfigured to combine the images pixel by pixel to provide a combinedimage. The combined image is a type of water heat map, the pixelsshowing a positive probability of water or ice. This may also bereferred to as “confidence”.

Subsequently, detection of the water in the resulting combined imagewill still take place. For example, water is detected while using athreshold, where pixels having a certain intensity above the thresholdare identified as water.

The device is configured to detect liquid or solid water on a surfacesuch as a road, for example.

In embodiments, the first optical bandwidth is selected from a spectralrange between 400 nm and 900 nm, and the second optical bandwidth isselected from a spectral range between 900 nm and 1200 nm. CMOS sensorsor silicon-based image sensors are sensitive up to ˜1000 nm, ortheoretically up to ˜1102 nm. The relatively broad spectral rangebetween ˜900 nm and ˜1000 nm is advantageous.

All aggregate states of water exhibit an increased absorption ratewithin the spectral range between 900 nm and 1200 nm as compared to thespectral range between 400 nm and 900 nm, whereas the absorption rate ofcommon road materials such as asphalt remains negligible within theentire spectral range between 400 nm and 1200 nm as compared to theabsorption peak of water, which is at ˜980 nm.

An aforementioned selection of the first optical bandwidth and thesecond optical bandwidth provides a difference in pixel intensities inregions comprising water between the pictures.

In embodiments, the first optical bandwidth has a value ranging between820 nm and 870 nm, 840 nm and 880 nm, or 850 nm and 890 nm at ahalf-width, and the second optical bandwidth has a value ranging between920 nm and 970 nm, 940 nm and 980 nm, or 950 nm and 990 nm at thehalf-width.

Optimum selection of the first optical bandwidth and of the secondoptical bandwidth provides a maximally possible difference in pixelintensities in regions comprising water between the pictures.

In embodiments, the optical sensor has a vertical polarization filterconfigured to reduce or eliminate the horizontally polarized light rays.

As long as the incident angle of the rays reflected off the watersurface is close to the Brewster angle, the rays reflected off the watersurface are mainly horizontally polarized, and the rays penetrating thewater are mainly vertically polarized. The rays that do not penetratethe water are not affected by the absorption of the water. A verticalpolarization filter removes or eliminates the horizontally polarizedrays. Since in the vicinity of the Brewster angle, most of thehorizontally polarized light undergoes specular reflection, a verticalpolarization filter may be used to remove specular reflection.

In embodiments, the optical sensor comprises a monochrome sensor or aCMOS sensor or a low-cost silicon sensor. These properties are notmutually exclusive. Monochromatic sensors comprise no filter layer or afilter layer that covers the entire area. Monochrome sensors are usedwhen filters are externally mounted. As soon as the filters are appliedin the form of different coatings, the sensor is no longer monochrome.

Common, inexpensive, silicon-based image sensors are referred to as CMOSsensors. In fact, CMOS is only one type of design. In embodiments, alltypes of design, such as CMOS or CCD, may be used. However, there arealso expensive CMOS sensors that are based on III-V semimetals. Inembodiments, low-cost silicon sensors are used, which have lowersensitivity than the more expensive type. Specific III-V semiconductorsensors, which are sensitive within higher wavelength ranges >1100 nm,are not necessary.

Examples of the present invention feature low-cost common CMOS imagesensors or less complex monochrome image sensors without Bayer filters.

In embodiments, the optical sensor comprises a first individual sensorcomprising a first optical bandpass filter having the first bandwidth,and a second individual sensor comprising a second optical bandpassfilter having the second bandwidth, the optical sensor being configuredto produce the first image by means of the first individual sensor andthe second image by means of the second individual sensorsimultaneously.

In further embodiments, the optical sensor comprises a third individualsensor having exchangeable optical bandpass filters, the exchangeableoptical bandpass filters comprising the first optical bandpass filterhaving the first bandwidth and the second optical bandpass filter havingthe second bandwidth, the optical sensor being configured to produce, insuccession, the first image by means of the third individual sensorcomprising the first optical bandpass filter and the second image bymeans of the third individual sensor comprising the second opticalbandpass filter.

In embodiments, the optical sensor comprises a fourth individual sensorcomprising a filter layer, the filter layer being configured to dividepixels of the image produced by the fourth individual sensor into twogroups in a row-wise, column-wise, or checkerboard-wise manner, whereinthe first optical bandpass filter having the first bandwidth is arrangedabove the pixels of the first group so that the pixels of the firstgroup produce the first image having the first optical bandwidth, andthe second optical bandpass filter having the second bandwidth isarranged above the pixels of the second group so that the pixels of thesecond group produce the second image having the second opticalbandwidth.

The optical sensor of the device is configured to provide the firstimage having the first bandwidth and the second image having the secondbandwidth as simultaneously as possible, so that, as far as possible,the same surface is imaged in the first image and in the second image.In embodiments, these requirements are met by several individual sensorscomprising several bandpass filters, by an individual sensor comprisingexchangeable optical bandpass filters, or by an individual sensorcomprising a filter layer.

In embodiments, the processor is configured to use to use, in combiningthe first image and the second image, pixel-by-pixel quotientcalculation, pixel-by-pixel difference calculation, pixel-by-pixelnormalized difference calculation, or difference calculation between 1and quotient calculation.

The processor of the device is configured to combine the pictures on apixel-by-pixel basis while using any of the calculations described aboveto produce a combined image. The combined image is a type of water heatmap. Each pixel of the combined image represents a positive probabilityvalue for water or ice in that pixel. This may also be referred to as“confidence”. In the case of a simple quotient, it may converselyrepresent the confidence of the dry surface in a pixel.

In alternative embodiments, individual pictures are no longer activelycombined, but raw data is passed directly to a trained model that makesdecisions on the basis of the raw data.

In embodiments, the processor is configured to provide a folded image.The processor is configured to scan the combined image with pixels on apixel-by-pixel basis or on a block-by-block basis by using a pixelwindow, to compute a weighted average of the values of the pixels of thepixel window, and to create a folded image from the weighted averages,wherein positions of the weighted averages in the folded imagecorrespond to positions of the pixel windows in the combined image. Thisstep increases the signal-to-noise ratio (SNR), where the signal is anabsorption difference and the noise includes all residuals.

A water absorption rate is about

${2. \cdot 10^{- 3}}{\frac{1}{mm}.}$This means that for a water film having a depth of 1 mm and a pixelsaturation of 255, only 2.0-10⁻³·255=0.51 at best will be absorbed.

For the absorption effect to become measurable at all, the adjacentpixels are also incorporated in a pixel window. The larger the pixelwindow, the more the absorption effect will be amplified and,accordingly, the resulting folded image or water heat map also becomesblurrier. For example, if a puddle consists of only a single pixel or oftoo few pixels, it will not be identified.

In embodiments, the processor is configured to compare values ofindividual pixels of the combined image or of a folded image derivedfrom the combined image to a threshold. The pixels having values thatexhibit a certain relationship to the threshold are detected as water. Athreshold decision is one of the simplest solutions for water detection.E.g., a better result may be achieved by a “feature detection”.

The combined image or the folded image is a kind of water heat map,where the pixels represent a probability of the appearance of water orice.

During localization or identification of water, values of the combinedimage are compared to a threshold pixel by pixel. That is, a finaldecision may be made with a threshold on a pixel-by-pixel basis or evenvia “feature extraction,” i.e., identification of contiguous areas whileusing a neural network, for example.

In embodiments, the device comprises an additional sensor configured toprovide information to the processor. The processor is configured todetect an aggregate state of the water while using the information inthe combined image.

The device comprising an additional sensor enables preventivelocalization and identification of wet areas, puddles, and ice or snowformations to deduce the actual danger or risk of traction loss, whichis a key feature for safe driving and future advanced driver assistancesystems (ADAS) or autonomous driving vehicles. Preventive hazarddetection of just 0.5 seconds before actual traction loss may enable anADAS to prevent a driver from making a wrong decision to panic or topreemptively deploy an airbag, thus preventing fatal injuries.

Absorption measuring alone cannot detect snow and cloudy ice. For this,the backscattering method provides localization, and the absorptionmeasurements are used as a confirmation. Thus, the sensor provides morethan just aggregate-state information.

In embodiments, the additional sensor comprises an additional opticalsensor. The additional optical sensor is configured to provide a thirdimage with a first polarization filter having a first polarization angleand a fourth image with a second polarization filter having a secondpolarization angle, the two polarization filters differing from eachother with regard to their polarization angles. Furthermore, theadditional optical sensor is configured to provide the third and fourthimages to the processor.

A combination of the absorption and backscattering measurements providesreliable features to assess the presence of water or ice on apixel-by-pixel basis.

Analysis of the polarization characteristics of the reflection of liquidwater shows a high polarization ratio of light with a predominantlyhorizontal orientation, while the ice crystals scatter the light ray andcause rotation of the wave. Ice crystals yield a more scatteredpolarization with a slightly shifted orientation.

The differences in polarization properties make it possible to implementa distinction between water and ice.

The different polarization planes of the images allow extracting thebackscattering properties, namely the phase, major axis and minor axisof a polarization ellipse. The angular offset of the polarization planesis not relevant here and may be chosen arbitrarily.

In embodiments, the additional optical sensor has an individual sensorcomprising a polarization filter layer.

The polarization filter layer is configured to divide picture elementsof the picture produced by the individual sensor into two groups in arow-by-row, column-by-column or checkerboard manner. Above the pictureelements of the first group, the first polarization filter is arrangedso that the picture elements of the first group produce the third imagewith the first polarization angle. The second polarization filter isarranged above the picture elements of the second group, so that thepicture elements of the second group produce the fourth image with thesecond polarization angle.

The additional optical sensor comprising the polarization filter layeris configured to produce the third image with the first polarizationfilter and the fourth image with the second polarization filter assimultaneously as possible, so that, as far as possible, the samesurface is imaged in the pictures.

In embodiments, the polarization angles of the polarization filters arearranged to be offset from one other by 90°. Two polarization filtershaving polarization angles of 0° and 90° are sufficient. Threepolarization filters having polarization angles offset from one anotherby 60° provide higher accuracy.

Uniform distribution of the polarization plane allows the major andminor axes of a polarization ellipse to be extracted in detail, thusmaking it easier to distinguish between water and ice.

In embodiments, the processor is configured to detect regions comprisingwater in the combined image or in a folded image derived from thecombined image. Furthermore, the processor is configured to evaluatepicture elements from the third and fourth images in the regionscomprising water, and to detect an aggregate state of the water, i.e.,whether the regions comprising water comprise water, liquid water, orice.

A combination of the absorption and backscattering measurements providesreliable features to assess the presence of water or ice on apixel-by-pixel basis. Regions comprising water are detected by anabsorption measurement, and the aggregate states of the water in theareas are assessed by using backscattering measurements.

In embodiments, the processor is configured to generate normalizedintensity values of the third and fourth images when evaluating thepicture elements of the third and fourth images.

Moreover, the processor is configured, when generating normalizedintensity values, to use a comparison of the values of the pictureelements with a sum of the values of the picture elements of the thirdand fourth images so as to detect the aggregate state of the water.

The processor is configured to detect an aggregate state of the water asbeing ice when the normalized intensity values of the third and fourthimages are within a range comprising +/−10% of the mean value of thenormalized intensity values of the third and fourth images, or to detectthe aggregate state of the water as being liquid water when a normalizedintensity value from the normalized intensity values of the third andfourth images is greater than 1.5 times the mean value of the normalizedintensity values of the third and fourth pictures.

The three images are normalized, on a pixel-by-pixel basis, to the meanvalue of the three images and are compared to one another. The simplethreshold decision that can be automated may be summarized as follows.

The ice surfaces exhibit large scattering and, thus, the pictureelements of all three images will have similar normalized intensityvalues. If the picture elements of all three images have similarnormalized intensity values in the same area, then picture elements ofthe area will be identified as being ice.

Reflection of liquid water shows a high polarization ratio of light withpredominantly horizontal orientation. If the normalized intensities inone area of an image are much higher than the intensities of the otherimages in the same area, picture elements of the area will be identifiedas being water.

In embodiments, the device includes a temperature sensor configured toprovide an ambient temperature value to the processor, which allows theprocessor to evaluate an aggregate state of the water.

An additional temperature sensor allows further complexities to bereduced. If the ambient temperature is far above or far below thefreezing point of the water, the region comprising water will beidentified as being water or ice, depending on the ambient temperature.

In embodiments, the processor is configured to sense a set of statevalues comprising information from the first image and the second image,and to provide a variable threshold as a function of the set while usingan artificial-intelligence element or a neural network. A neural networkmay be used to find a better threshold on a pixel-by-pixel basis or toidentify contiguous shapes in the complete picture, as a variant of“feature detection”.

In addition, the processor is configured to detect water in the combinedimage or an aggregate state of water in the combined image while usingthe variable threshold.

The variable threshold, which was created as a function of theinformation from the first image and from the second image while usingan artificial-intelligence element or a neural network, will yield abetter identification rate in different environmental situations.

In embodiments, the processor is configured to sense a set of statevalues comprising information from the third image and from the fourthimage, and to provide a variable threshold as a function of the setwhile using an artificial-intelligence element or a neural network.

In addition, the processor is configured to detect water in the combinedimage or an aggregate state of water in the combined image while usingthe variable threshold.

The variable threshold, which was created as a function of theinformation from the third image and from the fourth image while usingan artificial-intelligence element or a neural network, will yield abetter identification rate in different environmental situations.

Embodiments of the invention provide a means of locomotion comprisingthe device and an interface.

The device includes an additional sensor configured to provideinformation to the processor.

The processor of the device is configured to detect an aggregate stateof water while using the information in the combined image.

The interface is configured to alert a driver of the means of locomotionand/or to influence control of the means of locomotion if the devicedetects a solid state of the water.

The device of the means of locomotion enables safe driving by preventivelocalization and identification of wet areas, puddles, and ice or snowformations to deduce the actual danger or risk of traction loss.Preventive hazard detection prior to actual traction loss may enable anADAS to alert the driver or influence control of the means of locomotionvia the interface.

Further embodiments according to the present invention providecorresponding methods.

Embodiments according to the present invention will exhibit preventiveidentification of water and ice hazards on the road while using CMOSimage sensors.

With regard to the schematic figures shown, it shall be pointed out thatfunctional blocks shown are to be understood both as elements orfeatures of an inventive device and as corresponding method steps of aninventive method, and that corresponding method steps of the inventivemethod may also be derived therefrom.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be detailed subsequentlyreferring to the appended drawings, in which:

FIG. 1 shows a schematic representation of an embodiment of a devicecomprising an optical sensor and a processor;

FIG. 2 shows a schematic representation of an embodiment of a devicecomprising an optical sensor, an additional optical sensor and aprocessor;

FIG. 3 a shows a schematic representation of an embodiment of a means oflocomotion comprising the device and an interface;

FIG. 3 b shows a schematic representation of an embodiment of a means oflocomotion comprising the device;

FIG. 4 a shows a diagram of an absorption characteristic of water withinthe spectral range between 400 nm and 1000 nm;

FIG. 4 b shows a diagram of an absorption characteristic of water withinthe spectral range between 400 nm and 1200 nm;

FIG. 5 shows a schematic representation of an evaluation method of theabsorption measurement performed by the processor of the device;

FIG. 6 a shows an image without a filter over a surface comprisingwater;

FIG. 6 b shows a combined image over a surface comprising water;

FIG. 7 a shows non-polarized light rays partially penetrating the waterand partially being reflected off the water surface;

FIG. 7 b shows a diagram of the reflection coefficients of the lightrays polarized perpendicularly and in parallel at different incidentangles;

FIG. 7 c shows a diagram of the reflection coefficient rate

$\frac{{TE} - {TM}}{{TE} + {TM}}$of the light rays which are polarized vertically and in parallel and arereflected off the water surface at different incident angles;

FIG. 8 a shows three possible polarization ellipses defined by twopolarization planes;

FIG. 8 b shows a polarization ellipse defined by three polarizationplanes;

FIG. 8 c shows a polarization ellipse of liquid water with a largehorizontally polarized portion;

FIG. 8 d shows a polarization ellipse of solid water with uniformpolarization fractions;

FIG. 8 e shows a polarization ellipse of liquid water with no change inorientation;

FIG. 8 f shows a polarization ellipse of solid water with a change inorientation;

FIG. 9 a shows an image with a first polarization filter over a surfacewith water;

FIG. 9 b shows an image with a second polarization filter over a surfacewith water;

FIG. 9 c shows an image with a third polarization filter over a surfacewith water;

FIG. 9 d shows an image without a filter over a surface with water;

FIG. 10 shows an optical sensor comprising an image sensor and a filterlayer; and

FIG. 11 shows an implementation involving three different cameras, orlight-sensitive areas.

DETAILED DESCRIPTION OF THE INVENTION

Before embodiments of the present invention will be explained in detailbelow with reference to the drawings, it shall be pointed out thatidentical elements, objects and/or structures having the same functionor the same effect are provided with the same or similar referencenumerals in the different figures, so that the descriptions of theseelements which are given in the different embodiments areinterchangeable, or mutually applicable.

FIG. 1 shows a schematic representation of an embodiment of a device 100for identifying water on a surface 150, comprising an optical sensor 110and a processor 120 coupled to the optical sensor. The output of theprocessor 120 is the output of the device 100.

The optical sensor 110 includes a vertical polarization filter 115, afirst individual sensor 123 comprising a first optical bandpass filter133 and a second individual sensor 126 comprising a second opticalbandpass filter 136. The vertical polarization filter 115 is notrequired.

The first optical bandpass filter has a first bandwidth within which thewater has a first absorption rate. The second optical bandpass filterhas a second bandwidth within which the water has a second absorptionrate that is higher than the first absorption rate.

The device is configured to detect water on the surface 150. Light rays160, as will be discussed further below, partially penetrate the waterand are partially reflected off the water surface.

The vertical polarization filter 115 is configured to reduce oreliminate the horizontally polarized rays that are reflected off thewater surface in a specular manner.

The non-reduced rays will arrive at either the first individual sensor123 comprising the first optical bandpass filter 133 or at the secondindividual sensor 126 comprising the second optical bandpass filter 136.

The optical sensor 110 is configured to produce a first image 143 bymeans of the first individual sensor 123 and a second image 146 by meansof the second individual sensor as simultaneously as possible, so that,as far as possible, the same surface 150 is imaged in the first imageand in the second image. Furthermore, the sensor is configured toprovide the first image and the second image to the processor.

The processor is configured to provide a combined image 130 as acombination of the first image and the second image, and to detect orlocate water in the combined image, as will be discussed further below.The detected regions comprising water 140 are the outputs of theprocessor or the outputs of the device.

FIG. 2 shows a schematic representation of an embodiment of a device 200for identifying an aggregate state of the water on a surface 150 bymeans of an optical sensor 110, an additional optical sensor 210, and aprocessor 120 coupled to the optical sensor 110 and to the additionaloptical sensor 210. The output of the processor 120 is the output of thedevice 200.

The processor may be coupled to an optional additional temperaturesensor 230.

The optical sensor 110 includes an individual sensor 123 comprising afilter layer 233. As indicated in frontal view 233F, the filter layer isdivided into two groups in a row-by-row, column-by-column, orcheckerboard manner. The first optical bandpass filter having the firstbandwidth is located in the first group, and the second optical bandpassfilter having the second bandwidth is located in the second group.

The first optical bandpass filter has a first bandwidth within which thewater has a first absorption rate, and the second optical bandpassfilter has a second bandwidth within which the water has a secondabsorption rate that is higher than the first absorption rate.

The additional optical sensor 210 includes an individual sensor 243comprising a polarization filter layer 253. Similar to the filter layer233, the polarization filter layer 253 is divided into three groups in arow-by-row, column-by-column, or checkerboard manner. A firstpolarization filter having a first polarization angle is located in thefirst group, a second polarization filter having a second polarizationangle is located in the second group, and a third polarization filterhaving a third polarization angle is located in the third group.

The device 200 is configured to detect the aggregate state of the wateron the surface 150.

The optical sensor 110 is configured to produce a first image 143 havinga first bandwidth and a second image 146 having a second bandwidth.

The additional optical sensor 210 is configured to produce a third image263 with a first polarization filter, a fourth image 266 with a secondpolarization filter, and a fifth image 269 with a third polarizationfilter.

The optical sensor 110 and the additional optical sensor 210 areconfigured to provide the first, second, third, fourth, and fifth images143, 146, 263, 266, 269 to the processor 120.

The optional temperature sensor 230 is configured to provide an ambienttemperature value to the processor. If the ambient temperature is wellabove or well below the freezing point of water, the region comprisingwater will be identified as being water or ice, depending on the ambienttemperature.

The processor 120 is configured to evaluate an absorption measurement onthe basis of the first and second images 280 and to detect the water onthe surface 150. Furthermore, the processor 120 is configured toevaluate a backscattering measurement while using the third, fourth, andfifth images 290 and to detect an aggregate state of the water in theregions comprising water. The evaluation method 280 of an absorptionmeasurement and properties of the evaluation method 290 of abackscattering measurement will be further explained below.

The detected aggregate states 240 of the regions comprising water arethe outputs of the processor or the outputs of the device.

FIG. 3 a shows a schematic representation of an embodiment of a means oflocomotion 300 having a device 200 and an interface 310 coupled to thedevice.

As was explained in FIG. 2 , the device 200 is configured to detect anaggregate state in the regions comprising water on a surface 150.

The interface 310 is configured to alert a driver 320 of the means oflocomotion and/or to preemptively influence control of the means oflocomotion if the device detects a solid state of the water.

The device 200 of the means of locomotion 300 enables safe driving bypreemptively identifying and locating wet areas, puddles, and ice orsnow formations on the road and classifying the aggregate state when theambient temperature is near the freezing point of water. Inference of anactual hazard or risk of traction loss is a key feature for futureadvanced driver assistance systems (ADAS) or autonomous drivingvehicles.

Alternatively, FIG. 3 b shows a schematic representation of a secondembodiment of a means of locomotion 300 comprising a device 200. Thedevice 200 comprising an image sensor at a camera height of 1.40 m maylook ahead as far as between 5 m and 100 m. For example, if a devicewants to look ahead 100 m, the incident angle 330 will be approx. ˜1°.

FIG. 4 a shows a diagram 400 of the absorption characteristics 410 ofwater within the spectral range between 400 nm and 1000 nm. The spectralrange between 900 nm and 1000 nm has a high absorption rate and ishighlighted by a window 420.

FIG. 4 b shows a diagram 450 of an absorption characteristic 410 ofwater within the spectral range between 400 nm and 1200 nm. The spectralrange between 900 nm and 1200 nm has two absorption peaks. The device200 is sensitive within the wavelength range between 900 and 1000 nm.Specific III-V semiconductor image sensors sensitive in higherwavelength ranges >1100 nm are not necessary.

As was explained in FIG. 1 , the device 100 exploits the absorptioncharacteristics of water. The optical sensor 110 of the device 100 takestwo pictures 143, 146 over the same surface 150 by means of bandpassfilters 133, 136 for spectral ranges with high and low water absorptionrates, respectively. Regions comprising water in the first image 143 andthe second image 146 are imaged differently.

Since other environmental materials such as asphalt, for example,exhibit a consistent absorption rate within the spectral range between400 nm and 1200 nm, external influencing factors such as changingbackground or illumination, for example, will affect both picturesequally.

To reduce or eliminate the background, or to highlight the regionscomprising water, the processor 120 of the device 100 is configured tocombine the pictures 143, 146 with each other. By optimally selectingthe first optical bandwidth of the first optical bandpass filter 133 andthe second optical bandwidth of the second optical bandpass filter 136,a maximally possible difference in pixel intensities in the regionscomprising water between the pictures is provided.

The absorption peak of water is at about 980 nm, so a first bandpassfilter is used within the wavelength range between 900 and 1000 nm. Thesecond bandpass filter is actually freely selectable within thewavelength range between 400 and 900 nm.

In an optimum selection, the first optical bandwidth may have a valueranging between 820 nm and 870 nm, 840 nm and 880 nm, or 850 nm and 890nm, for example, at a half-width, and the second optical bandwidth mayhave a value ranging between 920 nm and 970 nm, 940 nm and 980 nm, or950 nm and 990 nm, for example, at the half-width.

FIG. 5 shows a schematic representation of an evaluation method 280 ofthe absorption measurement performed by the processor 120 of the device200 in FIG. 2 .

As a starting point, a surface 510 is shown which is partially coveredwith water 520, such as a road with a puddle.

As a first step, the optical sensor 110 of the device 200 produces afirst image 143 having a first optical bandwidth within which the water520 has a first absorption rate, and a second image 146 having a secondoptical bandwidth within which the water has a second absorption ratethat is higher than the first absorption rate.

Because of the different absorption rates, regions comprising water 520are imaged differently in the first image 143 and the second image 146.In the second image 146, regions comprising water 520 are considerablydarker.

As a second step, the processor 120 of the device 200 produces acombined image 530 from the first image 143 and from the second image146. In this case, pixel-by-pixel quotient calculation is used inpixel-by-pixel combining. Other methods of combining may includepixel-by-pixel difference calculation, pixel-by-pixel normalizeddifference calculation, or difference calculation between 1 and quotientcalculation.

The combined image or the result of the pixel-by-pixel quotientcalculation is a kind of heat map, so the more a quotient value deviatesfrom 1.0, the more likely it will be that water or ice there will bedetected there.

As a third step, the processor 120 of the device 200 detects or locatesregions comprising water. Locating or detecting the regions comprisingwater may be performed as a final decision with a threshold on apixel-by-pixel basis or via “feature extraction,” i.e., identificationof contiguous areas. The detected regions comprising water 140 are theoutputs of the evaluation method 280 of the device 200.

It should be noted here that a water absorption rate is about

${2. \cdot 10^{- 3}}{\frac{1}{mm}.}$That is, a 1 mm deep film of water with a pixel saturation of 255,absorption will be only 2.0·10⁻³·255=0.51 at best.

For the absorption effect to be measurable at all, the adjacent pixelsare also incorporated in a pixel window. The larger the pixel window,the more the absorption effect will be amplified and, accordingly, theblurrier the resulting folded image or water heat map will be. Forexample, if a puddle consists of only a single pixel or too few pixels,it will not be identified.

Furthermore, the difference between a simple difference calculation anda simple quotient calculation will be explained in more detail in thefollowing example.

As a starting point, a surface having light and dark areas is selectedwhich is covered by a water film. Within a spectral range within whichwater absorbs negligibly little, a measured value of e.g. 200 isobtained for the light area and e.g. 50 for the dark area. Within thespectral range within which water absorbs to a noticeable extent, onegets readings of, e.g., 0.8·200=160 for the light area and 0.8·50=40 forthe dark area.

The difference would now be 40 for the light area and 10 for the darkarea, which would lead to the wrong conclusion that there is more waterpresent on the light area.

However, the quotient results in the value of 0.8 for both areas. A dryarea corresponds to the value of 1.0 in the case of a quotientcalculation.

For example, pixel-by-pixel normalization of the difference would alsobe possible, e.g.

${\frac{{i_{1}\left( {x,y} \right)} - {i_{2}\left( {x,y} \right)}}{i_{1}\left( {x,y} \right)} = {1 - \frac{i_{2}\left( {x,y} \right)}{i_{1}\left( {x,y} \right)}}},$which would then simply be 1−the quotient. The values i₁(x,y) andi₂(x,y) represent individual pixel intensity values of the first and thesecond images at the position x, y.

FIG. 6 shows an exemplary scenario for the absorption measurementperformed by a device 100. FIG. 6 a shows the original surface, i.e., apicture without a filter, with a region comprising water 610 a.

FIG. 6 b shows a combined image or a folded image. The combined imagewas produced by pixel-by-pixel quotient calculation between a firstimage 143 with a spectral range between 900 nm and 1000 nm and a secondimage 146 with a spectral range between 800 nm and 900 nm. To be able toproduce a reduced picture, the pixels are averaged through a 5×5 pixelwindow.

In the example, the following bandpass filters were used:

-   -   850 nm CWL, 50 nm FWHM bandpass filter with a 400-2200 nm        vertical polarization filter with a contrast of 400:1, and    -   950 nm CWL, 50 nm FWHM bandpass filter with a 400-2200 nm        vertical polarization filter with a contrast of 400:1.

The combined image was evaluated with a simple threshold. The thresholddecision may be defined while using, for example, the followingfunction:max(round(i(x,y)−0.4),1.0)

Dry regions or regions of milky ice are light or white in the pictureand have a value greater than 0.9. Values less than 0.9 are dark orblack in the picture and are identified as water. Absorption ofclear/wet ice behaves like water.

Light may not penetrate milky ice or snow, so milky ice or snow has noeffect on the absorption rate.

Regions comprising water 610 b in the combined image are clearlyvisible. Furthermore, reflections of trees on the water surface hardlyhave an effect on the absorption measurement. The combined imagecorrects the measurements for changing external influences such assubsurface material, lighting, or a changing environment reflected inthe water body.

Clear ice behaves like water.

This absorption measurement method is configured to cope with unknownlight sources. The only requirement is that the light source has enoughenergy within the near infrared (NIR) range, i.e. below 1000 nm. Insunlight, there is no problem at all, and at night, car headlights suchas halogen lamps, for example, are sufficient.

The absorption peak of water is around 980 nm, so a first bandpassfilter is used within the wavelength range between 900 and 1000 nm.

The second bandpass filter is actually freely selectable within thewavelength range between 400 and 900 nm. In embodiments, the secondbandpass filter is within the wavelength range between 800 and 900 nm.

Thus, simply put, two colors are compared. However, the optical bandpassfilters 133, 136 that are used are not within the visible range. Theoptical sensors 110 used, e.g. CMOS sensors, have no Bayer filtercoating and are thus monochrome. The three colors red, green, blue(R-G-B) are nothing but bandpass filters, within the wavelength ranges˜600-700 nm, ˜500-600 nm, ˜400-500 nm.

FIG. 7 shows the reflection characteristics of water 760 detected by adevice 100. All forms of water exhibit significant specular reflection,which causes a polarization effect.

FIG. 7 a shows non-polarized light rays 710 partially penetrating water760 and partially being reflected off the water surface.

FIG. 7 b shows a diagram of the reflection coefficients of the lightrays that are polarized perpendicularly (TE) 753 and in parallel (TM)756 and that are reflected off the water surface, at different incidentangles 740. The incident angle perpendicular to the water surface has avalue of 0°. Furthermore, the Brewster angle 750, approx. 53° for water,is also marked in FIG. 7 b . With the Brewster angle 750, the reflectedrays are 100% horizontally polarized. As long as the incident angle 740of the reflected ray is close to the Brewster angle 750, the reflectedray 730 will be 100% horizontally polarized.

FIG. 7 c shows a diagram of the reflection coefficient rate

$\frac{{TE} - {TM}}{{TE} + {TM}}$of the light rays that are polarized vertically and in parallel and arereflected off the water surface, at different incidence angles 740. Thepolarization rate has a maximum value at the Brewster angle 750, i.e.,at about ˜53°.

In FIG. 7 a , the light rays 730 reflected off the water surface aremainly horizontally polarized, and the penetrating light rays 720 aremainly vertically polarized.

Only the penetrating light rays 720 are affected by the absorption ofthe water. The horizontally polarized rays 730 that are reflected in aspecular manner are not relevant to, or even interfere with, absorptionmeasurement. Reduction or elimination of the horizontally polarized rays730 that are reflected in a specular manner is possible with a verticalpolarization filter, such as the vertical polarization filter 115 inFIG. 1 .

As is shown in FIG. 7 b , the vertical polarization filter remainsadvantageous as long as the incident angle 740 of the reflected ray isclose to the Brewster angle 750, since here the reflected ray 730 is100% horizontally polarized. The further the incident angle is from theBrewster angle, the less effect the polarization filter will have.

If an image sensor of a means of locomotion, e.g. the means oflocomotion 300 in FIG. 3 , wants to look 100 m ahead at a camera heightof 1.40 m, the incident angle will be about ˜89°.

In the following, the backscattering measurement will be explained. Theevaluation 290 of the backscattering measurement is performed by theprocessor 120 of the device 200 while using the third, fourth, and fifthimages, the three images differing from one another with regard to theirpolarization planes.

The polarization angles of the images may be represented as apolarization ellipse. FIG. 8 a shows that the polarization ellipsecannot be determined unambiguously on the basis of two polarizationplanes 810, 820 since there are typically three possible solutions 830,840, 850. With the help of two polarization planes, it is still possibleto train a model that achieves a probable result.

FIG. 8 b shows three polarization planes 860, 870, 880 defining apolarization ellipse 890. The three polarization planes 860, 870, 880are evenly distributed, and the angular offset of the polarizationplanes is not relevant here.

FIGS. 8 c and 8 d show an evaluation 290 or classification of thebackscatter characteristic performed by the processor 120 of the device200 in FIG. 2 . In an evaluation 290 of the backscatteringcharacteristic, the relationship between the major and minor axes of apolarization ellipse is important. Here, two polarization filters, 0°and 90°, are sufficient in most cases. The third filter allows increasedaccuracy. The inclination of the ellipse depends strongly on the ambientlight and is therefore difficult to use.

As was explained in FIG. 7 a , the rays reflected off the water surfaceare mainly horizontally polarized. FIG. 8 c shows a polarization ellipseof liquid water with a large horizontally polarized proportion.

Ice crystals scatter the light ray and rotate the polarization ellipse.The result is a more scattered polarization with a slightly shiftedorientation. FIG. 8 d shows a polarization ellipse of solid water withuniform polarization components.

The rotation of the polarization ellipse depends on the illumination.With homogeneous illumination, the rotation of the polarization ellipseis also homogeneous and may be used as a feature.

FIG. 8 f shows that with homogeneous illumination, ice crystals scatterthe light ray and rotate the polarization ellipse. Meanwhile, FIG. 8 eshows a polarization ellipse of water without any change in orientation.

FIG. 9 shows a scenario exemplary for the backscattering measurementwith three exemplary representations and one raw picture. FIG. 9 d showsthe original surface, i.e., a picture without a filter. FIG. 9 d shows aroad surface with a dry area 920, a region comprising water 910, and twoicy surfaces 930.

FIGS. 9 a, 9 b, 9 c show the third, fourth, and fifth images,respectively, the three images differing from one another with regard totheir polarization planes. FIGS. 9 a-c are false-color representations.Here, the pictures from the polarization filters are placed on R (+60°),G (0°), B (−60°) to make the effect more vivid. Here, white (R˜G˜B) isice, green (R<G>B) is water, and purple (R>G<B) is dry.

The following polarization filters were used in the example:

-   -   400-2200 nm polarization filter with a contrast of 400:1 at 0°        (horizontal).    -   400-2200 nm polarization filter with a contrast of 400:1 at 60°.    -   400-2200 nm polarization filter with a contrast of 400:1 at        120°.

The backscattering measurement is independent of the absorptionmeasurement. Here, the three pictures taken are normalized to theaverage of the three pictures on a pixel-by-pixel basis and comparedwith one another. In normalization, the following equation is used:

${p_{1N} = \frac{p_{1}}{p_{1} + p_{2} + p_{3}}},$where p_(1N) represents a normalized pixel of the first image, and p₁,p₂ and p₃ represent individual picture-element values of the third,fourth and fifth images.

A simple threshold decision in this case would be, for example: ifp₀<<p₆₀ and p₀<<p₁₂₀, then water, or if p₀˜=p₆₀˜=p₁₂₀, then milkyice/snow, or if p₀>p₆₀˜=p₁₂₀, then clear ice. The values p₀, p₆₀ andp₁₂₀ represent individual picture-element values of the third, fourth,and fifth images.

In the first approaches, heat maps of the individual methods are simplycombined. The backscattering method measures surface scattering andwould interpret any smooth object as water. The absorption method wouldignore this.

What is at hand is liquid water only if the absorption rate is high andthe backscattering measurement ascertains little scattering or a smoothsurface.

In the case of clear ice, the logic is different. What is at hand isclear ice when the absorption rate is high, i.e. water is present, whilethe backscattering measurement shows high scattering.

Turbid ice/snow is poorly detected by absorption but is all the moreeasily detected by backscattering, since it exhibits no absorption butvery high scattering.

The absorption method may be improved by using algorithms to assessneighboring pixels and trained decision models for taking into accountall of the surroundings. To cover the different cases, a model istrained by means of machine learning here. For image sections of a sizeof 32×32 pixels, the recognition rate was at 98%, and was still at avery good 85% when cross-validated with field test data. When using aneural network that also evaluates the shape, and when performingtraining with field test data, more than 95% will again be achieved.

FIG. 10 shows a schematic representation of an embodiment of an opticalsensor 1000 comprising an image sensor 1020 comprising a filter layer1010. The filter layer 1010 is located above the image sensor 1020. Theimage sensor is divided into pixels 1030.

The filter layer is evenly divided into four different filter groups,F1, F2, F3, F4 on a pixel-by-pixel basis. For example, filter groups F1and F2 may include a bandpass filter, and filter groups F3 and F4 mayinclude a polarization filter,

Light rays 1040 will arrive at the filter layer 1010. Pixels 1030 ofimage sensor 1020 will receive filtered light rays from different filtergroups F1, F2, F3, F4 of the filter layer 1010. The image produced bythe individual sensor is divided into four pictures 1060, 1070, 1080,1090 by filter groups F1, F2, F3, F4, and the images are provided to theprocessor.

The optical sensor 1000 may replace the sensors 110 and 210 in thedevice 200.

Even though some aspects have been described within the context of adevice, it is understood that said aspects also represent a descriptionof the corresponding method, so that a block or a structural componentof a device is also to be understood as a corresponding method step oras a feature of a method step. By analogy therewith, aspects that havebeen described in connection with or as a method step also represent adescription of a corresponding block or detail or feature of acorresponding device. Some or all of the method steps may be performedby a hardware device (or while using a hardware device). In someembodiments, some or several of the most important method steps may beperformed by such a device.

FIG. 11 shows an implementation involving three different cameras, orlight-sensitive areas. A first camera 1101, a second camera 1102, and athird camera 1103 are arranged next to one another to sense the samelight illustrated by X_(i). The first camera 1101 has two filter layers1101 a and 1101 b. The second camera 1102 has two filter layers 1102 aand 1102 b, and the third camera 1103 has two filter layers 1103 a and1103 b. The filters or filter layers 1101 a and 1103 a are identical andfilter with regard to a first wavelength λ₁. The filter layer 1102 a orthe filter 1102 a filters with regard to a different wavelength λ₂.Moreover, the filters or filter layers 1101 b, 1102 b, and 1103 b arepolarization filters, the elements 1101 b and 1102 b of which areidentical and filter with regard to a first polarization ϕ1, whereas theelement 1103 b filters with regard to a second polarization ϕ2. Theresult of each camera and/or light-sensitive area 1101, 1102, 1103 isdepicted at 1104. Three images have been produced which each comprise adifferent combination of polarization (ϕ1, ϕ2) and absorption (λ₁ andλ₂).

For the evaluation method, ΔXφ and ΔXλ may be used as are present inbrackets on the left side of the block of equations 1105. In the4-camera/filter system, the sizes are directly obtained. In the present3-camera system shown in FIG. 11 , the sizes are additionally attenuatedby the further filter, as shown on the left side in the block ofequations 1105, since the sizes ΔXφ and ΔXλ are attenuated by the firstabsorption filter depicted by λ₁( ), and/or by the first polarizationfilter depicted by ϕ₁( ). However, since these two filters λ₁( ) and ϕ₁() are known, the expressions on the left side of the equations 1105 arecalculated back to the original responses ΔXφ and ΔXλ.

It shall be noted that it is not relevant whether or not the arrangementcomprises three dedicated cameras or one chip having 3 areas (pixels)and/or three areas 1102, 1102, 1103 per pixel.

In the advantageous embodiment in FIG. 11 , the following are used:

φ1=0° (horizontally polarized)

φ2=90° (vertically polarized)

λ1=850 nm (bandpass 50 nm FWHM)

λ2=950 nm (bandpass 50 nm FWHM)

The advantageous device thus includes the optical sensor having thelight-sensitive area (1101, 1102, 1103) for producing the first image(X₀₁) of the surface which has the first optical bandwidth (λ₁), and thesecond image (X₀₂) of the surface which has the second optical bandwidth(λ₂). Moreover, the additional optical sensor is configured to providethe third image (X₀₁) having the first polarization filter, and toprovide the fourth image (X₀₃) having the second polarization filter tothe processor. In addition, the processor is configured to determine adifference (X₀₁−X₀₂) from the first image and the second image, and tocalculate the combined image while using the first polarization filter.Furthermore, the processor is configured to determine a furtherdifference (X₀₁−X₀₃) from the third image and the fourth image, and tocalculate a polarization difference (ΔXλ) while using the firstabsorption filter so as to use the polarization difference (ΔXλ) forevaluating the regions comprising water.

The first image mentioned for the purpose of this application is, in thecase of the 3-camera/element solution described, the same image as thethird image. However, the second image and the fourth image aredifferent images of different light-sensitive areas 1102 and 1103. Inaddition, it shall be noted that the combined image thus cannotnecessarily be obtained by a simple difference but may be obtained byprocessing a difference, possibly while using a filter as in theembodiment of FIG. 11 .

Depending on particular implementation requirements, embodiments of theinvention may be implemented in hardware or in software. Theimplementation may be performed using a digital storage medium, forexample, a floppy disk, a DVD, a Blu-ray disc, a CD, a ROM, a PROM, anEPROM, an EEPROM, or a FLASH memory, a hard disk, or any other magneticor optical storage medium on which electronically readable controlsignals are stored, which may interact, or actually do interact, with aprogrammable computer system in such a way as to perform the respectivemethod. Therefore, the digital storage medium may be computer readable.

Thus, some embodiments according to the invention include a storagemedium having electronically readable control signals capable ofinteracting with a programmable computer system such that any of themethods described herein are performed.

Generally, embodiments of the present invention may be implemented as acomputer program product having program code, the program code beingoperative to perform any of the methods when the computer programproduct is running on a computer.

For example, the program code may also be stored on a machine-readablemedium.

Other embodiments include the computer program for performing any of themethods described herein, wherein the computer program is stored on amachine-readable medium.

In other words, an embodiment of the method of the invention is thus acomputer program comprising program code for performing any of themethods described herein when the computer program runs on a computer.

Thus, another embodiment of the methods according to the invention is adata carrier (or a digital storage medium or a computer-readable medium)on which the computer program for performing any of the methodsdescribed herein is recorded.

Thus, another embodiment of the method of the invention is a data streamor sequence of signals representing the computer program for performingany of the methods described herein. The data stream or sequence ofsignals may, for example, be configured to be transferred over a datacommunication link, such as over the Internet.

Another embodiment comprises a processing device, such as a computer orprogrammable logic device, configured or adapted to perform any of themethods described herein.

A further embodiment comprises a computer having installed thereon thecomputer program for performing any of the methods described herein.

Another embodiment according to the invention comprises a device orsystem configured to transmit a computer program for performing at leastone of the methods described herein to a receiver. The transmission maybe, for example, electronic or optical. The receiver may be, forexample, a computer, mobile device, storage device, or similar device.The device or system may include, for example, a file server fortransmitting the computer program to the receiver.

In some embodiments, a programmable logic device (for example, a fieldprogrammable gate array, an FPGA) may be used to perform some or all ofthe functionalities of the methods described herein. In someembodiments, a field programmable gate array may interact with amicroprocessor to perform any of the methods described herein. Ingeneral, in some embodiments, the methods are performed on the part ofany hardware device. This may be general-purpose hardware, such as acomputer processor (CPU), or hardware specific to the method, such as anASIC.

While this invention has been described in terms of several embodiments,there are alterations, permutations, and equivalents which fall withinthe scope of this invention. It should also be noted that there are manyalternative ways of implementing the methods and compositions of thepresent invention. It is therefore intended that the following appendedclaims be interpreted as including all such alterations, permutationsand equivalents as fall within the true spirit and scope of the presentinvention.

LITERATURE

-   A. L. Rankin, L. H. Matthies, and P. Bellutta (2011), “Daytime    water-detection based on sky reflections”, in Robotics and    Automation (ICRA 2011 IEEE International Conference on. IEEE, 2011,    pp. 5329-5336-   C. V. Nguyen, M. Milford, R. Mahony (2017): “3[) tracking of water    hazards with polarized stereo cameras”, 2017 IEEE International    Conference on Robotics and Automaton (ICRA) Singapore, May 29-Jun.    3, 2017-   V. Vikari, T. Varpula, and M. Kantanen (2009), “Road-condition    recognition using 24-GHz automotive radar”, IEEE Transactions on    Intelligent Transportation Systems, vol. 10, no. 4, pp. 639-648,    2009-   J. Casselgren, S. Rosendahl, M. Sjòdahl, P. Jonsson (2015), “Road    condition analysis using NIR illumination and compensating for    surrounding light”, in Optics and Lasers in Engineering 77(2016) pp    175-182-   A near-infrared optoelectronic approach to detection of road    conditions L. Colace, F. Santoni, G. Assanto, NooEL-Nonlinear Optics    and OptoElectronics Lab, University “Roma Tre”, Via della Vasca    Navale 84, 00146 Rome, Italy-   3D tracking of water hazards with polarized stereo cameras Chuong V.    Nguyen1, Michael Milford2 and Robert Mahony-   A Lane Detection Vision Module for Driver Assistance-   A. L. Rankin, L. H. Matthies, and P. Bellutta (2011), “Daytime    water-detection based on sky reflections”, in Robotics and    Automation (ICRA 2011 IEEE International Conference on. IEEE, 2011,    pp. 5329-5336-   C. V. Nguyen, M. Milford, R. Mahony (2017): “3[) tracking of water    hazards with polarized stereo cameras”, 2017 IEEE International    Conference on Robotics and Automaton (ICRA) Singapore, May 29-Jun.    3, 2017-   V. Vikari, T. Varpula, and M. Kantanen (2009), “Road-condition    recognition using 24-GHz automotive radar”, IEEE Transactions on    Intelligent Transportation Systems, vol. 10, no. 4, pp. 639-648,    2009-   J. Casselgren, S. Rosendahl, M. Sjòdahl, P. Jonsson (2015), “Road    condition analysis using NIR illumination and compensating for    surrounding light”, in Optics and Lasers in Engineering 77(2016) pp    175-182-   A near-infrared optoelectronic approach to detection of road    conditions L. Colace, F. Santoni, G. Assanto, NooEL-Nonlinear Optics    and OptoElectronics Lab, University “Roma Tre”, Via della Vasca    Navale 84, 00146 Rome, Italy-   3D tracking of water hazards with polarized stereo cameras Chuong V.    Nguyen1, Michael Milford2 and Robert Mahony-   A Lane Detection Vision Module for Driver Assistance Author(s):    Maček, Kristijan; Williams, Brian; Kolski, Sascha; Siegwart, Roland-   Daytime Water Detection Based on Color Variation Arturo Rankin and    Larry Matthies-   Daytime Water Detection Based on Sky Reflections Arturo L. Rankin,    Larry H. Matthies, and Paolo Bellutta-   Detecting water hazards for autonomous off-road navigation Larry    Matthies*, Paolo Bellutta, Mike McHenry-   Ice detection on a road by analyzing tire to road friction    ultrasonic noise D. Gailius, S. Jačėnas-   Ice formation detection on road surfaces using infrared thermometry    Mats Riehm, Torbjörn Gustaysson, Jörgen Bogren, Per-Erik Jansson-   Wet Area and Puddle Detection for Advanced Driver Assistance Systems    (ADAS) Using a Stereo Camera Jisu Kim, Jeonghyun Baek, Hyukdoo Choi,    and Euntai Kim*-   MULTISPECTRAL IMAGING OF ICE Dennis Gregoris, Simon Yu and Frank    Teti-   Near field ice detection using infrared based optical imaging    technology Hazem Abdel-Moati, Jonathan Morris, Yousheng Zeng, Martin    Wesley Corie II, Victor Garas Yanni-   New System for Detecting Road Ice Formation Amedeo Troiano, Eros    Pasero, Member, IEEE, and Luca Mesin-   Selection of optimal combinations of band-pass filters for ice    detection by hyperspectral imaging Shigeki Nakauchi, Ken Nishino,    and Takuya Yamashita-   A Sensor for the Optical Detection of Dangerous Road Condition    Armando Piccardi and Lorenzo Colace-   Polarization-Based Water Hazards Detection for Autonomous Off-road    Navigation Bin Xie, Huadong Pan, Zhiyu Xiang, Jilin Liu-   Road condition analysis using NIR illumination and compensating for    surrounding light Johan Casselgren, SaraRosendahl, MikaelSjödahl,    PatrikJonsson-   Self-Supervised Segmentation of River Scenes Supreeth Achar, Bharath    Sankaran, Stephen Nuske, Sebastian Scherer and Sanjiv Singh.

The invention claimed is:
 1. A device for identifying water on a surface, comprising: an optical sensor for producing a first image of the surface which exhibits a first optical bandwidth within which the water exhibits a first absorption rate, and a second image of the surface which exhibits a second optical bandwidth within which the water exhibits a second absorption rate that is higher than the first absorption rate; a processor for combining the first image and the second image to produce a combined image in which the surface is reduced or eliminated as compared to the water, and for detecting the water in the combined image; and an additional optical sensor configured to provide to the processor a third image with a first polarization filter comprising a first polarization angle, and to provide to the processor a fourth image with a second polarization filter comprising a second polarization angle, wherein the two polarization filters differ from each other with regard to their polarization angles, and wherein the processor is configured to detect an aggregate state of the water while using the third image and the fourth image in the combined image.
 2. The device as claimed in claim 1, wherein the first optical bandwidth comprises a waveband selected from a spectral range between 400 nm and 900 nm; and the second optical bandwidth comprises a waveband selected from a spectral range between 900 nm and 1200 nm.
 3. The device as claimed in claim 1, wherein the first optical bandwidth at a full width at half maximum exhibits a value ranging between 820 nm and 870 nm; and the second optical bandwidth at the full width at half maximum exhibits a value ranging between 950 nm and 990 nm.
 4. The device as claimed in claim 1, wherein the optical sensor comprises a vertical polarization filter configured to reduce or eliminate the horizontally polarized light rays.
 5. The device as claimed in claim 1, wherein the optical sensor comprises a silicon-based image sensor, a monochrome sensor, or a CMOS sensor.
 6. The device as claimed in claim 1, wherein the optical sensor comprises a first individual sensor comprising a first optical bandpass filter comprising the first bandwidth, and a second individual sensor comprising a second optical bandpass filter comprising the second bandwidth, wherein the optical sensor is configured to simultaneously produce the first image by means of the first individual sensor and the second image by means of the second individual sensor simultaneously; or wherein the optical sensor comprises a third individual sensor comprising exchangeable optical bandpass filters, the exchangeable optical bandpass filters comprising the first optical bandpass filter comprising the first bandwidth and the second optical bandpass filter comprising the second bandwidth, wherein the optical sensor is configured to produce, in succession, the first image by means of the third individual sensor comprising the first optical bandpass filter, and the second image by means of the third individual sensor comprising the second optical bandpass filter; or wherein the optical sensor comprises a fourth individual sensor comprising a filter layer, wherein the filter layer is configured to divide pixels of the image produced by the fourth individual sensor into two groups in a row-wise, column-wise or checkerboard-wise manner, wherein the first optical bandpass filter comprising the first bandwidth is arranged above the pixels of the first group, so that the pixels of the first group produce the first image comprising the first optical bandwidth, and the second optical bandpass filter comprising the second bandwidth is arranged above the pixels of the second group, so that the pixels of the second group produce the second image comprising the second optical bandwidth.
 7. The device as claimed in claim 1, wherein the processor is configured to use, in combining the first image and the second image, pixel-by-pixel quotient calculation, pixel-by-pixel difference calculation, pixel-by-pixel normalized difference calculation, or a difference between 1 and quotient calculation.
 8. The device as claimed in claim 1, wherein the processor is configured to produce a convolved image, wherein the processor is configured to scan the combined image with pixels on a pixel-by-pixel or block-by-block basis by using a pixel window, to compute a weighted average of the values of the pixels of the pixel window, and to produce a convolved image from the weighted averages, wherein positions of the weighted averages in the convolved image match positions of the pixel windows in the combined image.
 9. The device as claimed in claim 1, wherein the processor is configured to compare values of individual pixels of the combined image or of a convolved image derived from the combined image with a threshold, wherein the pixels with values that exhibit a certain relation to the threshold are detected as being water, or to perform feature detection to detect water.
 10. The device as claimed in claim 1, wherein the additional optical sensor comprises an individual sensor comprising a polarization filter layer, wherein the polarization filter layer is configured to divide picture elements of the image produced by the individual sensor into two groups in a row-by-row, column-by-column, or checkerboard manner, wherein the first polarization filter is arranged above the picture elements of the first group, so that the picture elements of the first group produce the third image with the first polarization angle, and the second polarization filter is arranged above the picture elements of the second group, so that the picture elements of the second group produce the fourth image with the second polarization angle.
 11. The device as claimed in claim 1, wherein the optical sensor and the additional optical sensor are configured in an individual sensor comprising a filter layer, wherein the filter layer is configured to divide picture elements of the image produced by the individual sensor into four groups in a row-by-row, column-by-column, or checkerboard manner, wherein the first optical bandpass filter comprising the first bandwidth is arranged above the pixels of the first group, so that the pixels of the first group produce the first image comprising the first optical bandwidth, wherein the second optical bandpass filter comprising the second bandwidth is arranged above the pixels of the second group, so that the pixels of the second group produce the second image comprising the second optical bandwidth, wherein the first polarization filter is arranged above the picture elements of the third group, so that the picture elements of the third group produce the third image with the first polarization angle, and wherein the second polarization filter is arranged above the picture elements of the fourth group, so that the picture elements of the fourth group produce the fourth image with the second polarization angle.
 12. The device as claimed in claim 1, wherein the polarization angles of the polarization filters are arranged to be offset from one another by 90°.
 13. The device as claimed in claim 1, wherein the processor is configured to detect regions comprising water in the combined image or in a convolved image derived from the combined image, to evaluate picture elements from the third and fourth images in the regions comprising water, and to detect an aggregate state of the water, whether the regions comprising water comprise liquid water or ice.
 14. The device as claimed in claim 13, wherein the processor is configured to generate normalized intensity values of the third and fourth images when evaluating the picture elements of the third and fourth images, to use, in generating the normalized intensity values, a comparison of the values of the picture elements with a sum of the values of the picture elements of the third and fourth images, to detect the aggregate state of the water, the processor being configured to detect an aggregate state of the water as ice when the normalized intensity values of the third and fourth images are within a range comprising +/−10% of the mean value of the normalized intensity values of the third and fourth images, or to detect the aggregate state of the water as liquid water when a normalized intensity value from the normalized intensity values of the third and fourth images is greater than 1.5 times the mean value of the normalized intensity values of the third and fourth images.
 15. The device as claimed in claim 1, comprising a temperature sensor configured to provide an ambient temperature value to the processor, by means of which the processor may evaluate an aggregate state of the water.
 16. The device as claimed in claim 1, wherein the processor is configured to sense a set of state values comprising information from the first image and the second image, and to provide a variable threshold as a function of the set while using an artificial-intelligence element or a neural network, and to detect, while using the variable threshold, water in the combined image or an aggregate state of the water in the combined image.
 17. The device as claimed in claim 1, wherein the processor is configured to sense a set of state values comprising information from the third image and from the fourth image, and to provide a variable threshold as a function of the set while using an artificial-intelligence element or a neural network, and to detect, while using the variable threshold, water in the combined image or an aggregate state of the water in the combined image.
 18. The device as claimed in claim 1, wherein the optical sensor comprises the light-sensitive area for producing the first image of the surface which exhibits the first optical bandwidth, and the second image of the surface which exhibits the second optical bandwidth, wherein the additional optical sensor is configured to provide the third image with the first polarization filter, and to provide the fourth image with the second polarization filter, wherein the processor is configured to determine a difference from the first image and the second image, and to calculate the combined image while using the first polarization filter, and wherein the processor is configured to determine a further difference from the third image and the fourth image, and to calculate a polarization difference while using the first absorption filter so as to use the polarization difference for evaluating the regions comprising water.
 19. Means of locomotion comprising: the device for identifying water on a surface, said device comprising: an optical sensor for producing a first image of the surface which exhibits a first optical bandwidth within which the water exhibits a first absorption rate, and a second image of the surface which exhibits a second optical bandwidth within which the water exhibits a second absorption rate that is higher than the first absorption rate; a processor for combining the first image and the second image to produce a combined image in which the surface is reduced or eliminated as compared to the water, and for detecting the water in the combined image; and an additional optical sensor configured to provide to the processor a third image with a first polarization filter comprising a first polarization angle, and to provide to the processor a fourth image with a second polarization filter comprising a second polarization angle, wherein the two polarization filters differ from each other with regard to their polarization angles, and wherein the processor is configured to detect an aggregate state of the water while using the third image and the fourth image in the combined image, and an interface, wherein the interface is configured to alert a driver of the means of locomotion and/or to influence control of the means of locomotion if the device detects a solid state of the water.
 20. A method of distinguishing a liquid or solid aggregate state of water in a region comprising water, the method comprising: acquiring a first image of the surface which exhibits a first optical bandwidth within which the water exhibits a first absorption rate, acquiring a second image of the surface which exhibits a second optical bandwidth within which the water exhibits a second absorption rate that is higher than the first absorption rate, combining the first image and the second image to acquire a combined image in which the surface is reduced or eliminated as compared to the water, detecting the water in the combined image, acquiring a third image with a first polarization filter comprising a first polarization angle, acquiring a fourth image with a second polarization filter comprising a second polarization angle, wherein the two polarization filters differ from each other with regard to their polarization angles, evaluating the picture elements of the regions comprising water of the third and fourth images, and detecting the aggregate state of the water on the basis of the evaluation of the picture elements.
 21. A non-transitory digital storage medium having a computer program stored thereon to perform the method of distinguishing a liquid or solid aggregate state of water in a region comprising water, the method comprising: acquiring a first image of the surface which exhibits a first optical bandwidth within which the water exhibits a first absorption rate, acquiring a second image of the surface which exhibits a second optical bandwidth within which the water exhibits a second absorption rate that is higher than the first absorption rate, combining the first image and the second image to acquire a combined image in which the surface is reduced or eliminated as compared to the water, detecting the water in the combined image, acquiring a third image with a first polarization filter comprising a first polarization angle, acquiring a fourth image with a second polarization filter comprising a second polarization angle, wherein the two polarization filters differ from each other with regard to their polarization angles, evaluating the picture elements of the regions comprising water of the third and fourth images, and detecting the aggregate state of the water on the basis of the evaluation of the picture elements, when said computer program is run by a computer. 